Flicker reduction method, image pickup device, and flicker reduction circuit

ABSTRACT

A system and method in which a fluorescent light flicker characteristic of an XY addressing type image pickup device such as a CMOS image pickup device is accurately detected and reliably and sufficiently reduced. This is achieved through signal processing without using a photosensitive element regardless of the level of a video signal of a subject and the type of a fluorescent lamp. A signal In′(x,y) is an RGB primary color signal or a luminance signal, each containing a flicker component. The signal In′(x,y) is integrated over a duration of time equal to or longer than one horizontal period, and a difference value between the integrated values of adjacent fields is normalized by the average value of the integrated values of three consecutive fields.

TECHNICAL FIELD

The present invention relates to a method for reducing a fluorescentlight flicker occurring in a video signal from an XY addressing typescanning image pickup element (an imager or an image sensor) such as aCMOS (complementary metal oxide semiconductor) image pickup element whena subject is photographed by the image pickup element under light of afluorescent lamp, an image pickup device, such as a video camera or adigital still camera employing the XY addressing type scanning imagepickup element such as the CMOS image pickup element, and a flickerreduction circuit for use in the image pickup device.

BACKGROUND ART

When a subject is photographed by a video camera under an illuminationof a fluorescent lamp directly operated from commercial AC power,chronological brightness level variations in a video signal as an imageoutput, i.e., a fluorescent light flicker occurs due to a differencebetween the frequency (twice the frequency of the commercial AC power)of a luminance change (light intensity change) of the fluorescent lampand a vertical synchronization frequency of the camera.

For example, in a zone where the frequency of the commercial AC power is50 Hz, a subject may be photographed by a CCD camera of an NTSC method(at a 60 Hz vertical synchronization frequency) under the illuminationof the non-inverter fluorescent lamp. In such a case as shown in FIG.28, one field frequency is 1/60 second while the period of the luminancechange of the fluorescent lamp is 1/100 second. The exposure timing ateach field drifts with respect to variations in the luminance of thefluorescent lamp, and an amount of exposure light at each pixel changesfrom field to field.

If the exposure time is 1/60 second, the amount of exposure light isdifferent within the same exposure time from duration a1 to duration a2to duration a3, and when the exposure time is shorter than 1/60 second(but not 1/100 second), the amount of exposure light time is differentwithin the same exposure from duration b1 to duration b2 to duration b3.

Since the exposure timing responsive to the luminance change of thefluorescent lamp reverts back to the original timing every three fields,the brightness level variation due to flickering is repeated every threefields. More specifically, the luminance ratio of each field (theappearance of a flicker) changes within an exposure period, but theperiod of flickering remains unchanged.

In a progressive camera such as a digital camera, the brightness levelvariation is repeated every three frames if the vertical synchronizationfrequency is 30 Hz.

To emit white light, a plurality of fluorescent lamps, for example, ared fluorescent lamp, a green fluorescent lamp, and a blue fluorescentlamp are typically used. Fluorescent materials for these lamps haveunique persistence characteristics thereof, and for a duration of timefrom a stop of discharging to a subsequent start of discharging, lightemission decays in accordance with the persistence characteristics.During this duration of time, light appearing white at first decayswhile the hue thereof changes at the same time. If the exposure timingdrifts, not only the brightness level variations but also the hue changeoccurs. Since the fluorescent lamp has unique spectral characteristicsthat a strong peak is present in a particular wavelength, a variablecomponent of a signal becomes different from color to color.

The color hue change and the difference in the variable component fromcolor to color lead to a so-called color flicker.

As shown in the bottom portion of FIG. 28, the amount of exposure lightremains constant regardless of the exposure timing if the exposure timeis set to be an integer multiple of periods ( 1/100 second) of theluminance variation of the fluorescent lamp. No flicker then occurs.

It is contemplated that illumination of the fluorescent lamp is detectedthrough signal processing of a camera in response to an operation of auser, and that the exposure time is set to be an integer multiple of1/100 second under the illumination of the fluorescent lamp. In thisarrangement, a simple method can fully control the generation of theflickering.

However, since this method does not allow the exposure time to be set toany value, the freedom of the exposure amount adjustment means forobtaining an appropriate amount of exposure is reduced.

A method for reducing the fluorescent light flicker under any shutterspeed (exposure time) is thus required.

An image pickup device having all pixels in one frame exposed at thesame exposure timing, such as a CCD image pickup device, offersrelatively easily such a method because the brightness level variationsand color variations due to the flickering appears only between fields.

If the exposure time is not 1/100 second, the flickering has arepetition frequency of three fields as shown in FIG. 28. To achieve aconstant average value of the video signal of each field, currentluminance variations and current color variations are predicted from thevideo signal three fields before, and the gain of the video signal ofeach field is adjusted based on the prediction results so that theflicker is reduced to a level that presents no problem in practice.

In the XY addressing type scanning image pickup device, such as a CMOSimage pickup device, however, the exposure timing drifts successivelyfrom pixel to pixel by one horizontal period of the reading clock (pixelclock) in a screen horizontal direction. Since all pixels are differentin the exposure timing, the above-referenced method cannot suppress theflickering.

FIG. 29 illustrates such flickering. The pixels successively drift inthe exposure timing in the screen horizontal direction as discussedabove. One horizontal period is sufficiently shorter than the period ofthe variation of the fluorescent light. Based on the assumption that thepixels on the same line have the same timing, the exposure timing ofeach line in a screen vertical direction becomes something like the oneshown in FIG. 29. This assumption presents no practical problem.

The exposure timing is different from line to line in the XY addressingtype scanning image pickup device, such as a CMOS image pickup device asshown in FIG. 29 (F1 represents such a drift in the exposure timing).Since the lines suffer from a difference in the amount of exposurelight, the brightness level variation and the color variation take placedue to the flickering not only between fields but also within eachfield. A strip pattern appears on a screen (with strips thereof alignedin the horizontal direction and the density of the stripes changing inthe vertical direction).

FIG. 30 illustrates an on-screen flicker if a subject is a uniformpattern. Since one horizontal period (one wavelength) of the subject is1/100 second, a stripe pattern of 1.666 periods appears in one frame.Let M represent the number of read lines per field, and one horizontalperiod of the stripe pattern corresponds to the number of read linesL=M*60/100. In this description and the drawing, an asterisk (*)represents a multiplication operation.

As shown in FIG. 31, three fields (three frames) correspond to fiveperiods (five wavelengths) of the stripe pattern, and if viewedcontinuously, the stripe pattern appears to drift in a verticaldirection.

FIG. 30 and FIG. 31 show only the brightness level variation due to theflicker. In practice, however, the above-described color variation alsoadditionally appears, thereby substantially degrading image quality. Thecolor flicker, in particular, becomes pronounced as the shutter speedbecomes fast. The XY addressing type scanning image pickup devicesuffers more from image quality degradation because the effect of thecolor flicker also appears on the screen.

If the exposure timing is set to be an integer multiple of periods (1/100 second) of the luminance variation of the fluorescent light, theamount of exposure becomes constant regardless of the exposure timing,and a fluorescent light flicker containing an on-screen flicker does notoccur.

With a variable electronic shutter speed feature incorporated, a CMOSimage pickup device becomes complex in structure. Even in an imagepickup device having the electronic shutter, the flexibility of theexposure amount adjusting means for achieving an appropriate exposure isreduced if only an integer multiple of 1/100 second is set as theexposure time to prevent flickering.

Methods for reducing the fluorescent light flickering for use in the XYaddressing type scanning image pickup device, such as the CMOS imagepickup device, have been proposed.

Patent document 1 (Japanese Unexamined Patent Application PublicationNo. 2000-350102) and patent document 2 (Japanese Unexamined PatentApplication Publication No. 2000-23040) discloses methods of estimatinga flicker component by measuring an amount of light of a fluorescentlamp with a photosensitive element or a measuring element andcontrolling a gain of a video signal from an image pickup element inresponse to the estimation result.

Patent document 3 (Japanese Unexamined Patent Application PublicationNo. 2001-16508) discloses another technique. In accordance with thedisclosed technique, two types of images are taken in two conditions,namely, a first electronic shutter value appropriate for a currentambient illumination condition and in a second electronic shutter valuehaving a predetermined relationship to a light and dark cycle of afluorescent lamp, a flicker component is estimated by comparing the twosignals, and a gain of a video signal from an image pickup device iscontrolled in response to the estimation results.

Patent document 4 (Japanese Unexamined Patent Application PublicationNo. 11-164192) discloses another technique. In accordance with thedisclosed technique, a brightness variation under an illumination of afluorescent lamp is recorded beforehand as a correction factor in amemory, the phase of a flicker component is detected from a video signalfrom an image pickup device taking advantage of a difference between thefrequency of a video signal component and the frequency of the flickercomponent, and the video signal is thus corrected in accordance with thecorrection factor in the memory in response to the detection results.

Patent document 5 (Japanese Unexamined Patent Application PublicationNo. 2000-165752) discloses another technique. In the disclosedtechnique, a correction coefficient is calculated from two video signalsthat are obtained as a result of exposures performed with a timedifference, the time difference causing the phase of flicker to beinverted by 180 degrees.

As disclosed in patent documents 1 and 2, the technique of estimating ofthe flicker component by measuring the amount of light of thefluorescent lamp with the photosensitive element or the measuringelement increases the size and the cost of the image pickup systembecause the photosensitive element or the measuring element is attachedto the image pickup device.

As disclosed in patent document 3, the technique of estimating theflicker component by photographing the two types of images in thedifferent shutter conditions (exposure conditions) requires a complexsystem in the image pickup device, and further this technique is notappropriate for taking a moving image.

The technique disclosed in patent document 4 uses the coefficientprepared beforehand in the memory as a correction signal. It ispractically impossible to prepare the correction coefficients for alltypes of fluorescent lamps. Depending on the type of the fluorescentlamp, detecting accurately the flicker component and reducing reliablythe flicker component are difficult. As disclosed in patent document 4,the technique of extracting the flicker component from the video signaltaking advantage of the difference between the frequencies of the videosignal component and the flicker component has difficulty in detectingthe flicker component distinctly from the video signal component in ablack background portion and a low-illuminance portion, each portionhaving a small amount of flicker component. If a moving image is presentin a screen, performance for detecting the flicker component issubstantially lowered.

As the technique disclosed in patent document 3, the technique disclosedin patent document 5 for estimating the flicker component byphotographing the two types of images at the different timings requiresa complex system in the image pickup device and is not appropriate fortaking a moving image.

In accordance with the present invention, a fluorescent light flickercharacteristic of an XY addressing type scanning image pickup devicesuch as a CMOS image pickup device is accurately detected and reliablyand sufficiently reduced through simple signal processing without usingan photosensitive element regardless of the level of a video signal of asubject and the type of a fluorescent lamp.

DISCLOSURE OF INVENTION

A flicker reduction method of a first invention for reducing afluorescent light flicker component in a video signal or a luminancesignal obtained by photographing a subject through an XY addressing typeimage pickup element under an illumination of a fluorescent lamp,includes

-   -   a step of integrating the video signal or the luminance signal,        as an input image signal, throughout a duration of time equal to        or longer than one horizontal period,    -   a step of normalizing the integrated value or a difference value        between the integrated values of adjacent fields or adjacent        frames,    -   a step of extracting a spectrum of the normalized integrated        value or the normalized difference value,    -   a step of estimating a flicker component from the extracted        spectrum, and    -   a step of performing a calculation operation on the estimated        flicker component and the input image signal to cancel out the        estimated flicker component.

A flicker reduction method of a second invention for reducing afluorescent light flicker component in each of color signals of colorsobtained by photographing a subject through an XY addressing type imagepickup element under an illumination of a fluorescent lamp, includes

-   -   a step of integrating the color signal of each color, as an        input image signal, throughout a duration of time equal to or        longer than one horizontal period,    -   a step of normalizing the integrated value or a difference value        between the integrated values of adjacent fields or adjacent        frames,    -   a step of extracting a spectrum of the normalized integrated        value or the normalized difference value,    -   a step of estimating a flicker component from the extracted        spectrum, and    -   a step of performing a calculation operation on the estimated        flicker component and the input image signal to cancel out the        estimated flicker component.

A flicker reduction method of a third invention for reducing afluorescent light flicker component in each of a luminance signal andeach of color signals of colors, obtained by photographing a subjectthrough an XY addressing type image pickup element under an illuminationof a fluorescent lamp, includes

-   -   a step of integrating each of the luminance signal and the color        signal of each color, as an input image signal, throughout a        duration of time equal to or longer than one horizontal period,    -   a step of normalizing the integrated value or a difference value        between the integrated values of adjacent fields or adjacent        frames,    -   a step of extracting a spectrum of the normalized integrated        value or the normalized difference value,    -   a step of estimating a flicker component from the extracted        spectrum, and    -   a step of performing a calculation operation on the estimated        flicker component and the input image signal to cancel out the        estimated flicker component.

In accordance with the flicker reduction method of the presentinvention, a signal component other than a flicker component is removedas the normalized integrated value or the normalized difference value,and a signal that allows the flicker component to be easily estimated ata high precision is obtained from a black background portion and alow-illuminance portion, each portion having a small amount of flickercomponent. By extracting the spectrum to an appropriate order from thenormalized integrated value and the normalized difference value, theflicker component is thus estimated at a high precision regardless ofthe type of a fluorescent lamp and a luminance varied waveform even inan area where a signal component becomes discontinuous by the effect ofa subject. The flicker component is reliably and sufficiently reducedfrom the input image signal by performing the calculation operation onthe estimated flicker component and the input image signal.

In particular, in accordance with the flicker component reduction methodof the second or third invention, the flicker component is detected fromthe color signal of each color, obtained as the video signal, on a percolor signal basis, or on a per luminance and color signal basis, andthe detected flicker component is reduced. The fluorescent light flickercontaining the color flicker is thus accurately detected and reliablyand sufficiently reduced.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system configuration of an image pickup device ofone embodiment of the present invention.

FIG. 2 illustrates one example of a digital signal processor of aprimary color system.

FIG. 3 illustrates one example of a digital signal processor of acomplementary color system.

FIG. 4 illustrates a first example of a flicker reducer.

FIG. 5 illustrates one example of a calculating block taking into asaturation region.

FIG. 6 illustrates a second example of the flicker reducer.

FIG. 7 illustrates a third example of the flicker reducer.

FIG. 8 illustrates one example of the flicker reducer under theillumination of a non-fluorescent lamp.

FIG. 9 illustrates another example of the flicker reducer under theillumination of a non-fluorescent lamp.

FIG. 10 illustrates an image pickup device that takes into account asubject that changes greatly in a short period of time in response to anoperation or an action of a photographer.

FIG. 11 illustrates one example of image pickup device that takes intoaccount the flicker component reduction process that becomes unnecessarydepending on a photographing condition.

FIG. 12 illustrates another example of image pickup device that takesinto account the flicker component reduction process that becomesunnecessary depending on a photographing condition.

FIG. 13 illustrates a basic configuration to adjust an estimated flickercomponent.

FIG. 14 illustrates a first specific example for adjusting the estimatedflicker component.

FIG. 15 illustrates a second specific example for adjusting theestimated flicker component.

FIG. 16 serves the purpose of explanation of FIG. 14 and FIG. 15.

FIGS. 17A and 17B illustrate equations serving the purpose ofexplanation of an example of adjusting the estimated flicker component.

FIGS. 18A and 18B illustrate equations serving the purpose ofexplanation of the example of adjusting the estimated flicker component.

FIGS. 19A and 19B illustrate equations serving the purpose ofexplanation of the example of adjusting the estimated flicker component.

FIGS. 20A-20E illustrate equations serving the purpose of explanation ofthe example of adjusting the estimated flicker component.

FIGS. 21A-21C illustrate equations serving the purpose of explanation ofthe example of adjusting the estimated flicker component.

FIGS. 22A and 22B serve the purpose of explanation of FIG. 8 and FIG. 9.

FIG. 23 illustrates a subject used in tests.

FIG. 24 is a plot of an integrated value of the subject of FIG. 23.

FIG. 25 is a plot of a difference value of the subject of FIG. 23.

FIG. 26 is a plot of a normalized difference value of the subject ofFIG. 23.

FIG. 27 is a plot of a flicker coefficient estimated for the subject ofFIG. 23.

FIG. 28 illustrates a fluorescent light flicker of a CCD image pickupdevice.

FIG. 29 illustrates a fluorescent light flicker in an XY addressing typeimage pickup element.

FIG. 30 illustrates a stripe pattern in one frame in the fluorescentlight flicker in the XY addressing type image pickup element.

FIG. 31 illustrates a stripe pattern in three consecutive frames in thefluorescent light flicker in the XY addressing type image pickupelement.

BEST MODE FOR CARRYING OUT THE INVENTION

[Embodiments of Image Pickup Devices: FIG. 1-FIG. 3]

(System Configuration: FIG. 1)

FIG. 1 illustrates a system configuration of one embodiment of an imagepickup device in accordance with the present invention. The image pickupdevice is here a video camera employing a CMOS image pickup element as aXY addressing type scanning image pickup -element.

In the image pickup device of this embodiment, namely, the video camera,light from a subject is incident on a CMOS image pickup device 12 via animage pickup optical system 11, and photoelectrically converted into ananalog video signal by the CMOS image pickup device 12. The resultinganalog video signal is thus obtained from the CMOS image pickup device12.

The CMOS image pickup device 12 includes, on a CMOS substrate, aplurality of two-dimensionally arranged pixels, each pixel including aphotodiode (a photo gate), a transfer gate (shutter transistor), aswitching transistor (address transistor), an amplification transistor,a reset transistor (reset gate), etc., and further a vertical scanningcircuit, a horizontal scanning circuit, and a video signal outputcircuit.

The CMOS image pickup device 12 may be a primary color system or acomplementary color system, and the analog video signal output from theCMOS image pickup device 12 is an RGB primary color signal or acomplementary color signal.

An analog signal processor 13 packaged in an IC (integrated circuit)processes the analog video signal from the CMOS image pickup device 12on a per color signal basis, thereby sample-holding the analog videosignal, gain controlling the analog video signal in an AGC (automaticgain control) process, and A/D converting the video signal into adigital signal.

The digital video signal from the analog signal processor 13 isprocessed by a digital signal processor 20 packaged in an IC, as will bediscussed later. In the digital signal processor 20, a flicker reducer25 reduces the digital video signal on a per signal component basisusing a method of the present invention as will be discussed later. Thedigital video signal is finally converted into a luminance signal Y, andred and blue color difference signals R-Y and B-Y to be output from thedigital signal processor 20.

A system controller 14, including a microcomputer, controls each blockin the camera.

More specifically, a lens drive control signal is supplied from thesystem controller 14 to a lens driver 15 packaged in an IC, and the lensdriver 15 drives lenses in an image pickup optical system 11.

The system controller 14 supplies a timing control signal to a timinggenerator 16. The timing generator 16 supplies a variety of timingsignals to a CMOS image pickup element 12 to drive the CMOS image pickupelement 12.

The system controller 14 receives detection signals for signalcomponents from the digital signal processor 20. The analog signalprocessor 13 gain controls the color signal in response to an AGC signalfrom the system controller 14. Signal processing of the digital signalprocessor 20 is controlled by the system controller 14.

If a subject changes in a short period of time in response to an actionof a photographer, the system controller 14 detects an output from ahand-shake sensor 17 connected to the system controller 14, and controlsa flicker reducer 25 as will be discussed later.

A control panel 18 a and a display 18 b, forming an interface 18, areconnected to the system controller 14 via an interface (I/F) 19 composedof a microcomputer. The system controller 14 detects a setting operationand a selection operation on the control panel 18 a. The systemcontroller 14 also displays a set status and a control statue of thecamera on the display 18 b.

If a subject changes greatly in a short period of time in response to acamera operation, such as a zoom operation, the system controller 14detects the camera operation, and controls the flicker reducer 25 aswill be discussed later.

If no flicker reduction process is required, the system controller 14detects that no flicker reduction is required and controls the flickerreducer 25 as will be discussed later.

(Primary Color System: FIG. 2)

FIG. 2 illustrates an example of the digital signal processor 20 of aprimary color system.

The primary color system is a three-panel system or a single panelsystem. The three-panel system includes the image pickup optical system11 of FIG. 1 including a light separation system for separating lightfrom a subject into the RGB primary colors, and a CMOS image pickupelement as the CMOS image pickup element 12 for the RGB colors. Thesingle-panel system includes, as the CMOS image pickup element 12, aCMOS image pickup element including RGB color filers periodicallyarranged on each pixel in a horizontal direction on the screen of alight input surface thereof. In this case, the RGB primary color signalsare read from the CMOS image pickup element 12.

In the digital signal processor 20 of FIG. 1, a clamp circuit 21 clampsa black level of each of the RGB primary color signals to apredetermined level, a gain adjustment circuit 22 gain adjusts theclamped RGB primary color signals in accordance with an amount ofexposure, and flicker reducers 25R, 25G, and 25B reduce flickercomponents in the gain adjusted RGB primary color signals in accordancewith a method of the present invention.

In the digital signal processor 20 of FIG. 2, a white balance adjustmentcircuit 27 adjusts the flicker-reduced RGB primary color signals, agamma correction circuit 28 corrects the white-balanced adjusted RGBprimary color signals, and a synthesis matrix circuit 29 generates theluminance signal Y, and the color difference signals R-Y and B-Y fromthe gamma-corrected RGB primary color signals.

The primary color system generates the luminance signal Y after theprocessing of the RGB primary color signals is complete as shown in FIG.2. Since the flicker component in the RGB primary color signals isreduced in the course of the process of the RGB primary color signals,the flicker component in each of the color components and the luminancesignal is sufficiently reduced.

Although flicker reducers 25R, 25G, and 25B are preferably arranged asshown in FIG. 2, the present invention is not limited to thisarrangement.

(Complementary color system: FIG. 3)

FIG. 3 illustrates one example of the digital signal processor 20 for acomplementary color system.

The primary color system is a single-panel system that includes, as theCMOS image pickup element 12 of FIG. 1, a CMOS image pickup elementhaving complementary color filters on a light input surface thereof. Thecomplementary color filter, as shown as a color filter 1 in FIG. 3, agreen color filter section 1G and a magenta color filter section 1Mg arealternately arranged on the pixels in a horizontal direction at onehorizontal line position Lo at every two lines, and a cyan color filtersection 1Cy and a yellow color filter 1Ye are alternately arranged onthe pixels in a horizontal direction at the other horizontal lineposition Le at every two lines.

In this case, the CMOS image pickup element 12 synthesizes and readsvideo signals of two adjacent horizontal line positions. At eachhorizontal period, a color signal synthesized from the green colorsignal and the cyan color signal and a color signal synthesized from themagenta color signal and the yellow color signal are alternatelyobtained at every pixel clock from the CMOS image pickup element 12.

In the digital signal processor 20 of FIG. 3, a clamp circuit 21 clampsa black level of the complementary color signal to a predeterminedlevel, a gain adjustment circuit 22 gain adjusts the clampedcomplementary color signal in response to an amount of exposure, aluminance synthesizing circuit 23 generates a luminance signal Y fromthe gain-adjusted complementary color signal, and a primary colorseparating circuit 24 separates the gain-adjusted complementary colorsignal into the RGB primary color signals.

In the digital signal processor 20 of FIG. 3, a flicker reducer 25Yreduces a flicker component in the luminance signal Y from the luminancesynthesizing circuit 23, and the flicker reducers 25R, 25G, and 25Breduce flicker components in the RGB primary color signals from theprimary color separating circuit 24 in accordance with a method of thepresent invention.

In the digital signal processor 20 of FIG. 3, a gamma correction circuit26 corrects the gradation of the flicker-reduced luminance signal,thereby resulting in a luminance signal Y as an output, a white balanceadjustment circuit 27 white-balance adjusts the flicker-reduced RGBprimary color signals, a gamma correction circuit 28 adjusts thegradation of the white-balance adjusted RGB primary color signals, and asynthesis matrix circuit 29 generates the color difference signals R-Yand B-Y from the gamma-corrected RGB primary color signals.

In the primary color system, the luminance signal and the RGB primarycolor signals are generated in a relatively earlier phase of the processof the digital signal processor 20 as shown in FIG. 3. This is becausethe luminance signal is easily generated from the synthesized signalthrough a simple additive process. If the RGB primary color signals aregenerated from the synthesized signal through a difference process, andthe luminance signal Y is generated from the RGB primary color signals,and an S/N ratio of the luminance signal is lowered.

If the luminance signal Y and the color signals are processed throughseparate lines, merely reducing the flicker component in each colorsignal is not sufficient in the reduction of the flicker component inthe luminance signal. The flicker components in each color component andthe luminance signal are sufficiently reduced by separately reducing theflicker component in the luminance signal as shown in FIG. 3.

The flicker reducers 25R, 25G, and 25B are preferably arranged as shownin FIG. 3, but the present invention is not limited to this arrangement.

[Embodiments of the Flicker Reduction Method: FIG. 4-FIG. 27]

The flicker reducers 25R, 25G, and 25B of FIG. 2, and the flickerreducers 25R, 25G, and 25B of FIG. 3 are constructed as discussed below.The flicker reducers 25R, 25G, and 25B are hereinafter collectivelyreferred to as a flicker reducer 25.

In the following discussion, an input image signal refers to a RGBprimary color signal or a luminance signal input to the flicker reducer25 prior to a flicker reduction process, and an output image signalrefers to a RGB primary color signal or a luminance signal, output fromthe flicker reducer 25, subsequent to the flicker reduction process.

In the following discussion, a subject is photographed by a CMOS cameraof an NTSC system (with a vertical synchronization frequency of 60 Hz)under an illumination of a fluorescent lamp operating from a commercialAC power source of 50 Hz. When the fluorescent light flicker is notreduced, the brightness level variations and the color variations takeplace not only between fields but also within each field and a strippattern of five periods (five wavelengths) appears over three fields(three wavelengths) on a screen as shown in FIGS. 29-31.

In a non-inverter fluorescent lamp as well as an inverter fluorescentlamp, flickering takes place if rectification is not sufficient. Thepresent invention is thus not limited to the non-inverter type.

(First Example of the Flicker Reduction Method: FIG. 4)

FIG. 4 illustrates a first example of the flicker reducer 25.

As shown in FIGS. 30 and 31, the subject is uniform. The flickercomponent is typically proportional to a signal intensity of thesubject.

Let In′(x,y) represent an input image signal (an RGB primary colorsignal or a luminance signal prior to the flicker reduction process) ofa subject at any pixel n(x,y) in any field n, and In′(x,y) is a sum of asignal component free from the flicker component and the flickercomponent proportional to the signal component as represented inequation (1) in FIG. 17A.

In(x,y) represents the signal component, Γn(y)*In(x,y) represents theflicker component, and Γn(y) represents a flicker coefficient. Onehorizontal period is sufficiently shorter than a light emission period (1/100 second) of a fluorescent lamp, and the flicker coefficient isconsidered as being constant on the same line in the same field, and theflicker coefficient is thus represented by Γn(y).

As represented by equation (2) in FIG. 17A, Γn(y) is expressed byFourier series for generalization. In this way, the flicker coefficientis expressed in a form that accounts for all of emission characteristicsand persistence characteristics, different from type to type offluorescent lamp.

In equation (2), λo represents the wavelength of the on-screen flickershown in FIG. 30. Let M represent the number of read lines per field,and λo is L (=M*60/100) lines. Here, ωo represents a normalizationangular frequency normalized by λo.

Here, γm represents an amplitude of the flicker component of each order(m=1, 2, 3, . . . ). Φmn represents an initial phase of the flickercomponent of each order, and is determined by the light emission period( 1/100 second) and the exposure timing of the fluorescent lamp. SinceΦmn reverts back to the same value every three fields, and a differenceof Φmn to the immediately preceding field is determined by equation (3)in FIG. 17A.

<Calculation and Storage of Integrated Value>

In the example of FIG. 4, an integrating block 31 integrates the inputimage signal In′(x,y) over one line in a horizontal direction on thescreen as shown in equation (4) of FIG. 17B to minimize the effect of apicture in the detection of a flicker. An integrated value Fn(y) is thuscalculated. As represented by equation (5) of FIG. 17B, αn(y) inequation (4) represents a value that is obtained by integrating a signalcomponent In(x,y) over one line.

The integrated value Fn(y) thus calculated is stored in an integratedvalue storage block 32 to detect flickering in subsequent fields. Theintegrated value storage block 32 is designed to store the integratedvalues of at least two fields.

If a subject is uniform, the integrated value αn(y) of the signalcomponent In(x,y) becomes constant. It is thus easy to extract a flickercomponent αn(y)*Γn(y) from the integrated value Fn(y) of the input imagesignal In′(x,y).

In a general subject, however, αn(y) contains m*ωo component, and it isimpossible to separate a luminance signal and a color component, as aflicker component, from a luminance signal and a color component as asignal component of the subject itself. It is impossible to extract onlythe flicker component. The flicker component in a second term issignificantly smaller than the signal component in a first term inequation (4), and the flicker component is almost buried in the signalcomponent.

FIG. 24 illustrates the integrated value Fn(y) in three consecutivefields of a subject (a color subject in practice) of FIG. 23 forreference. FIG. 24 is a plot of the integrated value Fn(y) of a redcolor, and Field: N+0 (solid line), Field: N+1 (dash line), and Field:N+2 (broken line) represent first, second, and third fields appearingconsecutively. As seen from FIG. 24, it is impossible to extract theflicker component directly from the integrated value Fn(y).

<Average Value Calculation and Difference Calculation>

As shown in FIG. 4, the integrated values of the three consecutivefields is used to remove the effect of αn(y) from the integrated valueFn(y).

In this example, during the calculation of the integrated value Fn(y),an integrated value Fn_1(y) on the same line in the first precedingfield immediately prior to the current field and an integrated valueFn_2(y) on the same line in the second preceding field immediately priorto the first preceding field are read from the integrated value storageblock 32. A average value calculating block 33 calculates an averagevalue AVE[Fn(y)] of the three integrated value Fn(y), Fn_1(y), andFn_2(y).

If the subject is identified as the same entity during the threeconsecutive fields, αn(y) may be assumed to have the same value. If themotion of the subject is small enough during the three consecutivefields, this assumption presents no problem in practice. In thecalculation of the average of the integrated value during the threeconsecutive fields, equation (3) means that signals with the phases ofthe flicker components thereof successively shifted by (−2π/3)*m aresummed. As a result, the flicker components cancel out each other. Theaverage value AVE[Fn(y)] is expressed by equation (6) of FIG. 18A.

The average value of the integrated values for the three consecutivefields is calculated based on the assumption that the approximation ofequation (7) of FIG. 18A holds. If the motion of the subject is large,the approximation of equation (7) does not hold.

In the flicker reducer 25 intended to work with a subject moving in alarge amount, the integrated value storage block 32 stores theintegrated values over at least three fields, and the average value ofthe integrated values of at least four fields including the integratedvalue Fn(y) of the current field. In this way, the operation of low-passfilters functioning in time axis reduces the effect of the movingsubject.

However, the flicker is repeated every three fields. To cancel out theflicker component, the average value of the integrated values of j (aninteger multiple of 3, equal to or greater than twice 3, namely, 6, 9, .. . ) fields needs to be calculated. The integrated value storage block32 is thus designed to store the integrated values of at least (j−1)fields.

FIG. 4 illustrates a case in which the approximation of equation (7) ofFIG. 18A holds. Furthermore in this example, a difference calculatingblock 34 calculates a difference between the integrated value Fn(y) ofthe current field from the integrating block 31 and the integrated valueFn_1(y) of the field immediately prior to the current field from theintegrated value storage block 32. The difference value Fn(y)−Fn_1(y)represented by equation (8) of FIG. 18B is thus calculated. Equation (8)also holds on condition that the approximation of equation (7) holds.

FIG. 25 illustrates the difference value Fn(y)−Fn_1(y) of the threeconsecutive fields of the subject of FIG. 23. Since the effect of thesubject is sufficiently removed from the difference value Fn(y)−Fn_1(y),the state of the flicker component (flicker coefficient) more distinctlyappears than in the integrated value Fn(y) of FIG. 24.

<Normalization of Difference Value>

Furthermore in the example of FIG. 4, the normalizing block 35normalizes the difference value Fn(y)−Fn_1(y) from the differencecalculating block 34 by dividing the difference value Fn(y)−Fn_1(y) bythe average value AVE[Fn(y)] from the average value calculating block33, thereby outputting the normalized difference value gn(y).

The normalized difference value gn(y) is expanded as represented byequation (9) of FIG. 19A using equation (6) of FIG. 18A and equation (8)of FIG. 18B and the sum-to-product formula of trigonometry. Thenormalized difference value gn(y) is also expressed by equation (10) ofFIG. 19B using the relationship of equation (3) of FIG. 17A. Here, |Am|and θm in equation (10) are respectively represented by equations (11a)and (11b).

Since the effect of the signal intensity of the subject still persistsin the difference value Fn(y)−Fn_1(y), the levels of luminancevariations and color variations due to the flickering are different fromarea to area. By normalizing the difference value gn(y) as describedabove, levels of the luminance variations and the color variations dueto the flickering are set to be consistent to the same level.

FIG. 26 illustrates the normalized difference value gn(y) of the threeconsecutive fields of the subject of FIG. 23.

<Estimation of the Flicker Component through Spectrum Extraction>

|Am| and θm respectively represented by equations (11a) and (11 b) ofFIG. 19B are an amplitude and an initial phase of the spectrum of eachorder of the normalized difference value gn(y). The normalizeddifference value gn(y) is Fourier transformed to determine the amplitude|Am| and the initial phase θm of the spectrum of each order. Equations(12a) and (12b) of FIG. 20A show that an amplitude γm and an initialphase Φmn of the flicker component of each order represented by equation(2) of FIG. 17A are thus obtained.

In the example of FIG. 4, a DFT block 51 discrete Fourier transformsdata of one wavelength (of L lines) of flickering of the normalizeddifference value gn(y) from the normalizing block 35.

Let DFT[gn(y)] represent a DFT operation, and Gn(m) represent the DFTresult of order m, and the DFT operation is expressed by equation (13)of FIG. 20B. Here, W in equation (13) is represented by equation (14).According to the definition of the DFT, the relationship betweenequations (11a) and (11b) and equation (13) is represented by equations(15a) and (15b) of FIG. 20C.

The amplitude γm and the initial phase Φmn of the flicker of each orderare thus determined from equations (12a), (12b), (15a), and (15b) withreference to equations (16a) and (16b) of FIG. 20D.

The data length of the DFT operation is set to the one wavelength (of Llines) of the flicker because a discrete spectrum group of an integermultiple of ωo can be directly obtained.

The FFT (Fast Fourier Transform) is typically used as a Fouriertransform for digital signal processing. However, the DFT isintentionally used in this embodiment of the present invention. The DFTis more convenient than the FFT because the data length of the Fouriertransform is not a power of 2. Alternatively, the FFT can also be usedby manipulating input data and output data.

The flicker component is sufficiently approximated under theillumination of an actual fluorescent lamp even if the order number islimited to m-th order. It is not necessary to output all data in the DFToperation. In comparison with the FFT, the DFT suffers from anyparticular drawback in terms of operation efficiency in the applicationof this invention.

The DFT block 51 extracts the spectrum through the DFT operation definedby equation (13), and then estimates the amplitude γm and the initialphase Φmn of the flicker component of each order through an operationrepresented by equations (16a) and (16b).

In the example of FIG. 4, a flicker generating block 53 calculates aflicker coefficient Γn(y) represented by equation (2) of FIG. 17A fromthe estimated values γm and Φmn from the DFT block 51.

As previously discussed, the flicker component is sufficientlyapproximated under the illumination of the light of the fluorescent lampeven if the order number is limited to the m-th order. In thecalculation of the flicker coefficient Γn(y) through equation (2), theorder of total sum is set to a predetermined order, such as a secondorder, rather than infinity.

FIG. 27 illustrates the flicker coefficient Γn(y) of the threeconsecutive fields of the subject of FIG. 23.

In the preceding method, the difference value Fn(y)−Fn_1(y) iscalculated and then normalized by the average value AVE[Fn(y)] in ablack background portion and a low-illuminance portion, where theflicker component is typically small and is entirely buried in thesignal component in the integrated value Fn(y). The flicker component isthus accurately detected.

The estimation of the flicker component from the spectrum to anappropriate order means that the approximation is effected with thenormalized difference value gn(y) incompletely reproduced. Even if adiscontinuity takes place in the normalized difference value gn(y), as aresult of an incomplete reproduction, depending on the state of thesubject, the flicker component of that portion is accurately estimated.

<Calculation for Flicker Reduction>

From equation (1) of FIG. 17A, the signal component In(x,y) having noflicker component is expressed in equation (17) of FIG. 20E.

In the example of FIG. 4, a calculating block 40 adds 1 to the flickercoefficient Γn(y) from the flicker generating block 53, and the inputimage signal In′(x,y) is divided by the resulting sum [1+Γn(y)].

The flicker component contained in the input image signal In′(x,y) isalmost entirely removed in this way. The calculating block 40 results ina signal component In(x,y) having no substantial flicker component as anoutput image signal (as the RGB primary color signal or the luminancesignal subsequent to flicker reduction).

If all above processes are not completed within the duration of time ofone field due to constraints on calculation performance of the system, afunction of storing the flicker coefficient Γn(y) over the three fieldsis provided in the calculating block 40 taking advantage of therepetition of the flickering every three fields. The stored flickercoefficient Γn(y) is calculated for the input image signal In′(x,y)subsequent to the three fields.

(Example Accounting for Saturation Region: FIG. 5)

If the calculating block 40 of FIG. 4 performs the calculationrepresented by equation (17) with the level of the input image signalIn′(x,y) falling within a saturation region, the signal component (acolor component or a luminance component) varies against the object ofthe invention. For this reason, the calculating block 40 preferably hasa structure of FIG. 5.

The calculating block 40 of FIG. 5 includes an adder circuit 41 foradding 1 to the flicker coefficient Γn(y) from the flicker generatingblock 53, a divider circuit 42 for dividing the input image signalIn′(x,y) by the sum [1+Γn(y)], a switch 43 on the input side, a switch44 on the output side, and a saturation level determining circuit 45.The saturation level determining circuit 45 determines on a per pixelbasis whether the level of the input image signal In′(x,y) is not lowerthan a threshold level of the saturation region.

If it is determined that the level of the input image signal In′(x,y) islower than the threshold level of the saturation region on each pixel,the saturation level determining circuit 45 sets the switches 43 and 44to the sides thereof to the divider circuit 42. As previously discussed,the result of the calculation of equation (17) is output from thecalculating block 40 as an output image signal.

If it is determined that the level of the input image signal In′(x,y) isnot lower than the threshold level of the saturation region on thepixel, the saturation level determining circuit 45 sets the switches 43and 44 to the sides thereof opposite from the divider circuit 42. Theinput image signal In′(x,y) is output as is as an output signal from thecalculating block 40.

When the level of the input image signal In′(x,y) falls within thesaturation region, the signal component (the color component or theluminance component) is free from variations, and a high-quality imagesignal results.

(Second Example of the Flicker Reduction Method: FIG. 6)

If the difference value Fn(y)−Fn_1(y) is normalized by the average valueAVE[Fn(y)] as shown in FIG. 4, a finite calculation accuracy iseffectively achieved. If calculation accuracy requirement is satisfied,the integrated value Fn(y) can be directly normalized by the averagevalue AVE[Fn(y)].

FIG. 6 illustrates such an example. The normalizing block 35 performs anormalization operation by dividing the integrated value Fn(y) from theintegrating block 31 by the average value AVE[Fn(y)] from the averagevalue calculating block 33, thereby outputting the normalized differencevalue gn(y).

The normalized difference value gn(y) is represented by equation (18) ofFIG. 21A. As represented by equation (19) of FIG. 21B, a subtractercircuit 36 subtracts 1 from the normalized difference value gn(y)represented by equation (18) and transfers the subtraction results tothe DFT (discrete Fourier transform) block 51 to make the subsequentprocess of the arrangement of FIG. 6 identical to the subsequent processof the arrangement of FIG. 4.

Since |Am|=γm and θm=Φmn, γm and Φmn are determined from equations (20a)and (20b) with reference to equations (15a) and (15b) of FIG. 20C.

In the example of FIG. 4, the DFT block 51 estimates the amplitude γmand the initial phase Φmn of the flicker component of each order throughthe calculation of equations (16a) and (16b) after extracting thespectrum through the DFT operation defined by equation (13). In theexample of FIG. 6, the DFT block 51 estimates the amplitude γm and theinitial phase Φmn of the flicker component of each order through thecalculation of equations (20a) and (20b) after extracting the spectrumthrough the DFT operation defined by equation (13). The subsequentprocess remains unchanged from that of the arrangement of FIG. 4.

Since the example of FIG. 6 does not need the difference calculatingblock 34, the flicker reducer 25 is simplified accordingly.

In this example as well, the calculating block 40 preferably has thestructure of FIG. 5.

(Third Example of the Flicker Reduction Method: FIG. 7)

If the approximation defined by equation (7) of FIG. 18A holds, theaverage value AVE[Fn(y)] for use in the normalization in the example ofFIG. 4 equals αn(y) defined by equation (6), and the second term[αn(y)*Fn(y)] of equation (4) of FIG. 17B significantly smaller than thefirst term αn(y) has an insignificant effect on the normalization.

The integrated value Fn(y) is used instead of the average valueAVE[Fn(y)] in the normalization without any particular problem. Theflicker component is effectively detected in the same manner as when theaverage value AVE[Fn(y)] is used.

In the example of FIG. 7, the normalizing block 35 performs anormalization operation by dividing the difference value Fn(y)−Fn_1(y)from the difference calculating block 34 by the integrated value Fn(y)from the integrating block 31. The subsequent process remains unchangedfrom that of the arrangement of FIG. 4.

In the example of FIG. 7, it is sufficient if the integrated valuestorage block 32 holds the integrated value of one field, and theaverage value calculating block 33 is not required. The flicker reducer25 is thus simplified in structure.

In this example as well, the calculating block 40 preferably has thestructure of FIG. 5.

(Example of Another Process Performed Under Non-Fluorescent Lamp: FIG. 8and FIG. 9)

No particular problem is presented in the above-referenced flickerreduction process when a photographing operation is performed under theillumination of a non-fluorescent lamp (under an non-fluorescentillumination environment). However, if an otherwise unnecessary processis performed, the effect on image quality becomes a concern even if theflicker component is low enough.

If a photographing operation is performed under the illumination of thenon-fluorescent lamp, the flicker reduction process is preferablydisabled. The flicker reducer 25 is preferably designed to output theinput image signal In′(x,y) as is as an output image signal from theflicker reducer 25.

FIG. 8 illustrates an example of such a flicker reducer 25. A normalizedand integrated value calculating block 30 is designed as shown in theexample of FIG. 4, FIG. 6 or FIG. 7. In the example of FIG. 4 and FIG.7, the difference value Fn(y)−Fn_1(y) rather than the integrated valueFn(y) is normalized, and for convenience of explanation, thecorresponding block is also referred to as a normalized and integratedvalue calculating block.

In the example of FIG. 8, a under-fluorescent-lamp determination block52 is arranged between the DFT block 51 and the flicker generating block53.

Under the illumination of the fluorescent lamp, the level (amplitude) γmof the calculated component of each order is sufficiently higher at m=1than a threshold Th and sharply lowers as m increases as shown in FIG.22A. Under the illumination of the non-fluorescent lamp, the level ofthe component is lower than the threshold Th at every order as shown inFIG. 22B.

The spectrum is ideally zero under the illumination of thenon-fluorescent lamp. In practice, however, the subject moves, and thenormalized difference value gn(y) or the normalized integrated valuegn(y)−1, generated from the signals of a plurality of consecutivefields, inevitably contains a tiny amount of frequency component.

The under-fluorescent-lamp determination block 52 thus determineswhether the level at m=1 is above the threshold Th. If it is determinedthat the level at m=1 is above the threshold Th, theunder-fluorescent-lamp determination block 52 determines that thephotographing operation is performed under the illumination of thefluorescent lamp. The estimated values of γm and Φmn from the DFT block51 are directly output to the flicker generating block 53. In this case,the above-described flicker reduction process is executed.

If the level of the component at m=1 is lower than the threshold Th, theunder-fluorescent-lamp determination block 52 determines that thephotographing operation is performed under the non-fluorescent lamp, andsets the estimated value of γm of all order m to be zero. The flickercoefficient Γn(y) becomes zero, and the input image signal In′(x,y) isoutput as is as an output image signal from the calculating block 40.

FIG. 9 illustrates such an example. In this example, as in the exampleof FIG. 8, the under-fluorescent-lamp determination block 52 determineswhether the photographing operation is performed under the under theillumination of the fluorescent lamp. If it is determined that thephotographing operation is performed under the illumination of thenon-fluorescent lamp, the under-fluorescent-lamp determination block 52sets a detection flag COMP_OFF, stops the process of the flickergenerating block 53 and the calculating block 40, and outputs the inputimage signal In′(x,y) as is as an output image signal from thecalculating block 40. If it is determined that the photographingoperation is performed under the illumination of the fluorescent lamp,the under-fluorescent-lamp determination block 52 resets the detectionflag COMP_OFF and the flicker reduction operation is thus performed.

If the photographing operation is performed under the illumination ofthe non-fluorescent lamp in the example of FIG. 9, not only an adverseeffect on image quality is prevented but also power consumption isreduced.

(Subject Greatly Changing in Response to an Operation or an Action of aPhotographer: FIG. 10)

A subject occasionally changes in a short period of time in response tozooming, panning, tilting, or hand-shaking of a photographer. In such acase, the assumption of equation (7) of FIG. 18A fails to hold. As aresult, the flicker detection accuracy is degraded.

The image pickup device is constructed as shown in FIG. 10 taking intoconsideration a subject that changes greatly in a short period of timein response to an operation or an action of a photographer.

As shown in FIG. 10, switches 55 and 56 and a flicker storage block 57are arranged between the flicker generating block 53 and the calculatingblock 40 additionally to the flicker reducer 25 of FIG. 4, FIG. 6, orFIG. 7. The detection flag DET_OFF, to be discussed later, output fromthe system controller 14 is fed to the switches 55 and 56 as a switchingsignal.

The flicker storage block 57 stores the flicker coefficient Γn(y) of thethree fields. Each time the process of one field is complete, theflicker coefficient Γn(y) is stored for use in the next field. Theoutput read from the flicker storage block 57 is switched every threefields.

The system controller 14 detects a subject if the subject changesgreatly in a short period of time in response to the operation or theaction of the photographer.

For example, if the photographer presses a zoom key on the control panel18 a, the system controller 14 detects the pressing via the interface19. The system controller 14 controls lenses in response to thephotographer's zoom operation such as a telephoto and a wide-angleoperation. The hand-shaking of the photographer is detected by thehand-shake sensor 17, and the information of the hand-shaking is sent tothe system controller 14. In response to the hand-shake information, thesystem controller 14 performs anti-shake control. Upon detecting panningor tilting, the system controller 14 lightens the degree of anti-shakecorrection during panning, for example. Such control techniques remainunchanged from those that are performed in ordinary cameras.

In the example of FIG. 10, the system controller 14 sets the detectionflag DET_OFF if the operation or the action of the photographer causingthe subject to be changed greatly in a short period of time is detected.The system controller 14 resets the detection flag DET_OFF if neither ofsuch operation and action is performed.

In a normal state under which the subject does not change greatly in ashort period of time, the detection flag DET_OFF is reset. In theflicker reducer 25, the switch 55 is set to the side thereof to theflicker generating block 53. The flicker coefficient Γn(y) is fed fromthe flicker generating block 53 to the calculating block 40. The flickerreduction operation is performed. With the switch 56 turned on, theflicker coefficient Γn(y) is stored in the flicker storage block 57.

The detection flag DET_OFF is set if the subject changes greatly in ashort period of time in response to the operation or the action of thephotographer. In the flicker reducer 25, the switch 55 is set to theside thereof to the flicker storage block 57. The calculating block 40receives the flicker coefficient Γn(y) at a high detection precisionlevel immediately prior to the operation or the action of thephotographer, read from the flicker storage block 57, instead of theflicker coefficient Γn(y) at a low detection precision level. Theflicker reduction operation is performed. The switch 56 is turned off,and thus prevents the flicker coefficient Γn(y) at the low detectionprecision level from being stored onto the flicker storage block 57.

The flicker detection accuracy is heightened even when the subjectchanges greatly in a short period of time in response to the operationor the action of the photographer. The flicker is reliably andsufficiently reduced.

Furthermore, the detection flag DET_OFF is supplied to the normalizedand integrated value calculating block 30, the DFT block 51, and theflicker generating block 53. If the subject changes greatly in a shortperiod of time in response to the operation or the action of thephotographer, the detection flag DET_OFF is set. The set detection flagDET_OFF stops the process of each of the normalized and integrated valuecalculating block 30, the DFT block 51, and the flicker generating block53. In this case, power consumption is also reduced.

In this case, the flicker coefficient Γn(y) is replaced with theimmediately prior flicker coefficient thereof. A more front stagesignal, for example, an integrated value Fn(y), may be replaced with theimmediately prior signal thereof.

(Example of Another Process Performed Depending on PhotographingCondition: FIG. 11 and FIG. 12)

As will be discussed later, the flicker reduction operation becomesunnecessary depending on the photographing condition. In view of theeffect on image quality, performing otherwise unnecessary flickerreduction operation is undesirable when the photographing operation isperformed under the illumination of the non-fluorescent lamp asdescribed above.

First, the photographing conditions that require no flicker reductionoperation include the case in which a video camera or a digital stillcamera, each capable of taking both a moving image and a still image,photographs a still image.

A camera employing an XY addressing type image pickup element such as aCMOS image pickup element can set exposure timings of all pixels in oneframe (including exposure start timings and exposure end timings) to beuniform, and can control the effect of the fluorescent light flicker. Areading operation from the image pickup element is free from thelimitation of frame rate that is applied to moving image capturing, andis thus performed at a slow speed in a light blocked state with amechanical shutter closed.

Based on a camera operation on the control panel 18 a in the embodimentof FIG. 1, the system controller 14 determines whether to take a stillimage with the exposure of all pixels in one frame set to the sametiming.

Second, the photographing states that need no flicker reductionoperation also include the case in which a photographing operation isperformed outdoors under the light of the sun, and the case in which theexposure time (electronic shutter time) is set to be an integer multipleof periods ( 1/100 second) of luminance changes of the fluorescent lightby adjusting the amount of exposure.

Whether the photographing operation is performed under the illuminationof the fluorescent lamp is determined by referencing the level of thespectrum extracted by the DFT block 51 as previously discussed withreference to FIGS. 8 and 9. The photographing operation performedoutdoors under the light of the sun is categorized as a photographingoperation under the illumination of the non-fluorescent lamp, and inthis case, the system controller 14 directly determines from the amountof light of a subject that a photographing operation is performed underthe illumination of the non-fluorescent lamp.

As previously discussed, the camera employing the XY addressing typeimage pickup element such as a CMOS image pickup element is free fromthe fluorescent light flicker if the exposure time is set to be aninteger multiple of the periods ( 1/100 second) of the luminance changesof the fluorescent lamp. The system controller 14 directly detectswhether the exposure time is set to be an integer multiple of periods ofthe luminance change of the fluorescent lamp by adjusting the exposureamount.

If the system controller 14 determines that the photographing staterequires no flicker reduction operation, the flicker reduction operationis not effected. The system is designed so that the input image signalIn′(x,y) is output as is as an output image signal from the flickerreducer 25.

FIG. 11 illustrates one example of such a system configuration. In thisexample of the flicker reducer 25, a nulling block 59 is arrangedbetween the DFT block 51 and the flicker generating block 53. Thenulling block 59 is controlled by a flicker reduction on/off controlsignal from the system controller 14.

If the system controller 14 determines that the flicker reductionoperation is required, the flicker reduction on/off control signal isset to an on state, and the nulling block 59 outputs the estimatedvalues of γm and Φmn from the DFT block 51, as are, to the flickergenerating block 53. In this case, the flicker reduction operation iseffected.

If the system controller 14 determines that the flicker reductionoperation is unnecessary, the flicker reduction on/off control signal isset to an off state. The nulling block 59 nulls the estimated value ofγm of the order m to zero. In this case, the flicker coefficient Γn(y)also becomes zero, and the input image signal In′(x,y) is output as isas an output image signal from the calculating block 40.

FIG. 12 illustrates still another example. The calculating block 40 inthe flicker reducer 25 includes the adder circuit 41, the dividercircuit 42 and the switches 43 and 44, all shown in FIG. 5, but does notinclude the saturation level determining circuit 45 of FIG. 5. Theswitches 43 and 44 are controlled for switching by the flicker reductionon/off control signal from the system controller 14.

If the system controller 14 determines that the flicker reductionoperation is necessary, the switches 43 and 44 are set to the sidethereof to the divider circuit 42. As previously discussed, thecalculation result of equation (17) is output as an output image signalfrom the calculating block 40.

If the system controller 14 determines that no flicker reductionoperation is necessary, the switches 43 and 44 are set to the sidesthereof opposite from the divider circuit 42, and the input image signalIn′(x,y) is output as is as an output image signal from the calculatingblock 40.

In the example of FIG. 12, the flicker reduction on/off control signalis fed to the normalized and integrated value calculating block 30, theDFT block 51, and the flicker generating block 53. If the systemcontroller 14 determines that no flicker reduction operation isnecessary, the process of each of the normalized and integrated valuecalculating block 30, the DFT block 51 and the flicker generating block53 is stopped. Power: consumption is thus reduced in this example.

(Adjustment of Detected Flicker Component: FIG. 13-FIG. 15)

In accordance with each of the above-referenced methods, the flickercomponent in the input image signal is reliably and effectively reducedwhen the fluorescent light flicker is steadily and regularly generated.

However, in each of the above-referenced methods, the averagecalculation or the difference calculation is performed throughout aplurality of fields to detect the flicker component. For this reason,the flicker component cannot be accurately detected in a transitionaland unstable state, for example, at the moment the fluorescent lamp isswitched on or off or at the moment of any person's entrance into orexit out of the room illuminated by a fluorescent lamp. If the flickerreduction operation is effected in accordance with the flicker componentobtained in such a transitional and unstable state, an undesirablecorrection can be performed on the input image signal.

A variation in an angle of field in a horizontal direction can be causedin response to a horizontal motion of a subject, a camera operation suchas panning and zooming, and hand shaking of a photographer. Suchvariations are reliably and stably reduced. However, flicker reductionperformance is slightly lowered for a variation in the angle of view ina vertical direction caused in response to a vertical motion of thesubject, the camera operation such as panning and zooming, and handshaking of the photographer.

This problem is eliminated by the method illustrated in FIG. 10. Inaccordance with this method, however, the flicker component Γn(y) isswitched between when the subject is stable free from a large variationin a short period of time and when the subject changes greatly in ashort period of time in response to an operation or an action of thephotographer. For this reason, the photographer may feel the image ofthe subject discordant.

The effect of external disturbance cannot be controlled even in a normalstable state. In the method of FIG. 10, the image pickup device respondsdirectly to the external disturbance because of the fast trackingfeature thereof, and suffers from an error in the flicker reductionagainst the object of the flicker reduction operation.

To reduce the effect of external disturbance, a filtering operationusing an LPF (low-pass filter) is performed in the course of estimatingthe flicker component, and a time constant thereof is prolonged to slowthe tracking feature in the flicker estimation.

However, the setting of slow tracking feature also results in a slowtracking feature in the above-referenced transitional state. The flickerreduction operation cannot be performed quickly although the flickerreduction operation is required, for example, at the moment thefluorescent lamp is switched on or at the moment of any person'sentrance into the room illuminated by the fluorescent lamp. Conversely,the flicker reduction operation is continuously performed although theflicker reduction operation is required no longer, for example, at themoment the fluorescent lamp is switched off or at the moment of anyperson's exit out of the room illuminated by the fluorescent lamp.

Instead of performing the calculation operation on the detected flickercomponent and the input image signal, the amplitude and the phase of thedetected flicker component are adjusted as necessary, and then theadjusted flicker component and the input image signal are subjected tothe calculation operation. In this way, the flicker reduction operationis flexibly and properly performed in a variety of cases.

In the following example, parameters related to the flicker reductionoperation, for example, the amplitude and the phase of the estimatedflicker component are adjusted.

FIG. 13 illustrates a basic structure of such an arrangement. In thisexample, the system controller 14 captures data of the amplitude γm andthe initial phase Φmn of the estimated flicker component from the DFTblock 51 in the flicker reducer 25. A parameter controller 14 a in thesystem controller 14 adjusts the data as described later, and inputs theadjusted data of the amplitude γm′ and initial phase Φmn′ to the flickergenerating block 53 in the flicker reducer 25.

The flicker generating block 53 calculates the flicker coefficientΓn(y), represented by equation (2) of FIG. 17A, from the adjusted dataof the amplitude γm′ and initial phase Φmn′ instead of the amplitude γmand initial phase Φmn obtained from the DFT block 51. In this example,γm and Φmn are replaced with γm′ and Φmn′ in equation (2) of FIG. 17A.

As shown in FIG. 13, the normalized and integrated value calculatingblock 30 in the flicker reducer 25 has the same structure as thecounterpart in FIG. 4. Alternatively, the normalized and integratedvalue calculating block 30 may have the same structure as thecounterpart shown in FIG. 6 or FIG. 7.

<First Specific Example: FIG. 14>

FIG. 14 illustrates a first specific example of the system controller14.

The number of lines for data of the amplitude γm and the initial phaseΦmn from the DFT block 51 are m per field in practice, but a single lineonly is shown. The same is true of the output data of the amplitude γm′and the initial phase Φmn′ from the parameter controller 14 a.

The data of the amplitude γm and the initial phase Φmn from the DFTblock 51 are fed to digital LPFs (low-pass filters) 61 and 62, theoutput data of the digital LPF 61 is fed to a gain adjusting circuit(multiplier circuit) 63, and the output data of the gain adjustingcircuit 63 is input to the flicker generating block 53 as adjustedamplitude γm′. The output data of the digital LPF 62 is input to theflicker generating block 53 as adjusted initial phase Φmn′.

A time constant Ta of the digital LPF 61 and a time constant Tp of thedigital LPF 62 are set by the time constant setting block 65. A gain(multiplication coefficient) Ka of the gain adjusting circuit 63 is setby a gain setting block 66.

Preferably, the time constants of the digital LPFs 61 and 62 arecontinuously set to any value within a constant range. If a desired timeconstant cannot be set, a time constant close to the desired one may beset. If a single LPF cannot continuously change the time constant, aplurality of LPFs having separate time constants may be internallyarranged, and by providing time constants Ta and Tp as a control code tothe LPFs, a single LPF may be selected from the plurality of LPFs.

The initial phase Φmn periodically varies during the generation offlickering. For example, if the frequency of the commercial AC powersource is 50 Hz and the vertical synchronization frequency of the camerais 60 Hz, the initial phase Φmn becomes the same value every threefields, and a difference of the initial phase Φmn takes place from theimmediately prior field as represented by equation (3) of FIG. 17A.

The digital LPF 62 must serve as one LPF for the data of the same phasetaking into consideration the variation in the initial phase Φmn. Morespecifically, if the variation period of the initial phase Φmn is threefields as described above, three LPFs are arranged as the digital LPF62, and the data of the initial phase Φmn is distributed among the threeLPFs.

A state detecting block 68 receives the data of the amplitude γm and theinitial phase Φmn, and AE (automatic exposure) control information andAWB (automatic white balance) control information, obtained in thesystem controller 14. More specifically, the AE control information ison-screen brightness information, and the AWB control information iscolor temperature information and information as to whether the camerais illuminated under a fluorescent lamp.

In response to these inputs, the state detecting block 68 detects thephotographing conditions affecting the generation of the fluorescentlight flicker. More specifically, the state detecting block 68determines whether a current photographing condition is under theillumination of the fluorescent lamp, or a transitional state from theillumination of the non-fluorescent lamp to the illumination of thefluorescent lamp when the fluorescent lamp is turned on or atransitional state from the illumination of the fluorescent lamp to theillumination of the non-fluorescent lamp when the fluorescent lamp isturned off. In response to the determination results, a control mode isdetermined.

A control mode indicating signal is fed to the time constant settingblock 65 and the gain setting block 66 to indicate the determinedcontrol mode thereto. In response, the time constant setting block 65sets the time constants Ta and Tp in the digital LPFs 61 and 62,respectively, and the gain setting block 66 sets the gain Ka in the gainadjusting circuit 63.

FIG. 16 lists determination criteria for the state detection of thestate detecting block 68. If the flickering occurs steadily andregularly under the illumination of the fluorescent lamp, the amplitudeγm of the estimated flicker component becomes generally constant, andthe initial phase Φmn takes substantially the same value every constantperiod (every three fields at a commercial power source frequency of 50Hz and a camera vertical synchronization frequency of 60 Hz).

These inputs are sufficient to cause the state detecting block 68 todetermine that the flicker is generated steadily and regularly under theillumination of the fluorescent lamp.

Since the on-screen brightness varies generally periodically under theillumination of the fluorescent lamp, the brightness information for theAE control is sufficient to cause the state detecting block 68 todetermine that the camera is under the illumination of the fluorescentlamp.

In the AWE control, a light source is estimated from detected colorinformation, and a determination of whether the light source is afluorescent lamp is determined based on the estimated light source. Adetermination of whether the camera is under the illumination of thefluorescent lamp can be performed based on the light source estimatinginformation for the AWB control.

The above-referenced information over a plurality of past fields isgenerally checked to enhance detection accuracy.

Upon determining that the flickering takes place steadily and regularlyunder the illumination of the fluorescent lamp, the state detectingblock 68 sets, as the control mode, a mode A to be discussed later.

If no flickering takes place regularly under the illumination of thenon-fluorescent lamp, the amplitude γm of the estimated flickercomponent contains only a noise component, and varies randomly in thevicinity of zero, and the initial phase Φmn randomly varies because ofthe noise.

Such information is sufficient for the state detecting block 68 todetermine that no flicker reduction operation is necessary under theillumination of the non-fluorescent lamp.

From the brightness information for the AE control indicating thatvariations in the screen brightness have no periodicity under theillumination of the non-fluorescent lamp, the state detecting block 68determines that the camera is under the illumination of thenon-fluorescent lamp. From the light source estimating information forthe AWB control, the state detecting block 68 determines that the camerais under the illumination of the non-fluorescent lamp.

In this example, the above-referenced information over a plurality ofpast fields is generally checked to enhance detection accuracy.

Upon determining that no flickering takes place steadily and regularlyunder the illumination of the non-fluorescent lamp (that no flickerreduction operation is necessary), the state detecting block 68 sets, asthe control mode, a mode B to be discussed later.

In response to the control mode determined by the state detecting block68, the time constant setting block 65 sets the time constants Ta and Tpof the digital LPFs 61 and 62, and the gain setting block 66 sets thegain Ka of the gain adjusting circuit 63. Actual values to be set aredetermined as described below in response to the system configurationand requirements to the system.

The time constant Ta of the digital LPF 61 is discussed first. Aspreviously discussed, the amplitude γm of the estimated flickercomponent slightly varies in the vicinity of zero and is generallyconstant in both the mode A (in which the flickering takes placesteadily and regularly under the illumination of the fluorescent lamp)and the mode B (in which no flickering takes place regularly under theillumination of the non-fluorescent lamp). If external disturbance isapplied, the amplitude γm is not constant.

To make the system robust to the external disturbance, the time constantTa of the digital LPF 61 is preferably set to be longer. However, if thecontrol mode is transitioned from the mode A to the mode B or from themode B to the mode A, the time constant Ta of the digital LPF 61 ispreferably set to be shorter to achieve a fast tracking feature duringtransitions.

More specifically, the amplitude γm must satisfy two mutuallycontradictory requirements at the same time. The method of FIG. 4, FIG.6, or FIG. 7 provides an algorithm originally robust to the externaldisturbance.

In practice, a shorter time constant Ta is set with a view to thetracking feature. Most preferably, dynamic control may be introduced toprovide a longer time constant Ta in either the mode A or the mode B anda shorter time constant Ta in a transition from the mode A to the mode Bor in a transition from the mode B to the mode A.

The time constant Tp of the digital LPF 62 is discussed now. During themode A (when the flickering takes place steadily and regularly under theillumination of the fluorescent lamp), the initial phase Φmn becomessubstantially the same value every constant period based on theprinciple of the flicker generation as shown in FIG. 16. A sufficientlylonger time constant Tp needs to be set to be robust to the externaldisturbance.

In contrast, the initial phase Φmn takes a random number during the modeB (when no flickering takes place regularly under the illumination ofthe non-fluorescent lamp), and no particular advantage is provided bythe setting of a longer time constant Tp. More specifically, during themode B, the time constant Tp may be set to any value taking intoconsideration the effect of a gain adjustment to be discussed later.

The time constant Ta and the time constant Tp may be alternated betweenthe mode A and the mode B.

The gain Ka of the gain adjusting circuit 63 is now discussed. Duringthe mode A (when the flickering takes place steadily and regularly underthe illumination of the fluorescent lamp), the amplitude γm issubstantially constant as shown in FIG. 16, and the gain Ka can be setto 1.

The gain Ka determines a correction rate of the amplitude γm (if a gainKa=1 is set, 100% of the input is output with a correction rate atzero), and the correction rate of the amplitude γm is directlycontrolled by varying the gain Ka.

Under actual photographic environments, the amplitude may beintentionally enlarged or intentionally decreased. For this reason, thegain Ka is not limited to 1, and the system may be designed to set thegain Ka to be larger than 1 or smaller than 1.

During the mode B (in the state that no flicker is generated steadilyunder the illumination of the non-fluorescent lamp), the amplitude γmtakes a random value in the vicinity of zero in response to noise. Sinceno flicker reduction operation is required by nature during the mode B,the gain Ka is set to zero to disable the unnecessary process.

The mode A and the mode B as the steady state (the state that theflicker is steadily generated or the state that no flicker is steadilygenerated) have been discussed. If the mode detected by the statedetecting block 68 is transitioned from the mode A to the mode B, it islikely that the photographic environment has changed from under theillumination of the fluorescent lamp to under the illumination of thenon-fluorescent lamp. If the mode detected by the state detecting block68 is transitioned from the mode B to the mode A, it is likely that thephotographic environment has changed from under the illumination of thenon-fluorescent lamp to under the illumination of the fluorescent lamp.

As previously discussed with reference to FIG. 4, FIG. 6, or FIG. 7, inaccordance with the basic method of the present invention, the flickercomponent is extracted through the averaging operation or the differenceoperation throughout a plurality of fields. During the transition, oneportion of a signal string for use in the averaging operation or thedifference operation contains a flicker component and another portion ofthe signal string has no flicker component.

As a result, an error takes place in the flicker component obtainedthrough the averaging operation or the difference operation, and anerror also takes place in the detected amplitude γm and initial phaseΦmn. If the flicker coefficient Γn(y) is calculated from theerror-affected amplitude γm and initial phase Φmn, the output imagesignal is also adversely affected.

To alleviate this problem, the gain setting block 66 detects atransitional state and controls the gain Ka in response to thetransitional state.

More specifically, if the control mode is shifted from the mode A to themode B, the reliability of each of the amplitude γm and the initialphase Φmn has dropped at the start of the transition. Immediatelysubsequent to the transition, the gain Ka is switched from 1 to zero,and the flicker reduction operation of the flicker generating block 53and the calculating block 40 is stopped. Alternatively, the gain Ka isgradually lowered, and the flicker reduction operation of the flickergenerating block 53 and the calculating block 40 is smoothly stopped.

Conversely, if the control mode is transitioned from the mode B to themode A, the reliability of each of the amplitude γm and the initialphase Φmn is still low at the start of the transition. After time elapseuntil the reliability of each of the amplitude γm and the initial phaseΦmn rises to a sufficiently high level, the flicker generating block 53and the calculating block 40 performs the flicker reduction operation.Alternatively, the gain Ka is gradually raised so that the flickergenerating block 53 and the calculating block 40 smoothly perform theflicker reduction operation.

<Second Specific Example: FIG. 15>

FIG. 15 illustrates a second specific example of the system controller14.

In this example, memories 71-74, switches 75-78, and a state detectingblock 69 are added to the example of FIG. 14.

The memory 71 stores the data of the amplitude γm, the memory 72 storesthe data of the initial phase Φmn, the memory 73 stores the output dataof the gain adjusting circuit 63, and the memory 74 stores the outputdata of the digital LPF 62. The switches 75-78 select the input data andthe output data of the memories 71-74, respectively, in response to thedetection results of the state detecting block 69. The output data ofthe switch 75 is fed to the digital LPF 61, the output data of theswitch 76 is fed to the digital LPF 62, the output data of the switch 77is fed to the flicker generating block 53 as the data of the amplitudeγm′, and the output data of the switch 78 is fed to the flickergenerating block 53 as the data of the initial phase Φmn′.

The state detecting block 69 receives zooming information and hand-shakeinformation. In response to the zooming information, the state detectingblock 69 determines whether a large variation in the angle of view takesplace in response to a zooming action, and in response to the hand-shakeinformation, the state detecting block 69 determines whether a largevariation in the angle of view takes place in response to panning andtilting actions and a large-amplitude hand shake.

If it is determined that no large variations take place in the angle ofview, the state detecting block 69 sets the switches 75-78 to the sidesthereof opposite from the sides of the memories 71-74. Normally, theflicker reduction operation is performed as in the example of FIG. 14.

If it is determined that a large variation takes place in the angle ofview, the state detecting block 69 sets the switches 75-78 to the sidesthereof to the memories 71-74.

Since the reliability of each of the amplitude γm and the initial phaseΦmn drops when a large variation occurs in the angle of view, amplitudedata and initial phase data acquired in the past and respectively storedin the memory 73 and the memory 74 are input to the flicker generatingblock 53 as the amplitude γm′ and the initial phase Φmn.

Since the amplitude γm and the initial phase Φmn are steady during themode A (in the state that the flickering is generated steadily andregularly under the illumination of the fluorescent lamp) as shown inFIG. 16, the use of the past data presents no problem at all. On thecontrary, the past data should be positively used.

However, if low-reliability data is successively input to the digitalLPFs 61 and 62 in the middle of variation in the angle of view, theamplitude γm′ and the initial phase Φmn′ obtained immediately subsequentto the setting of the switches 77 and 78 to the sides thereof oppositefrom the memories 73 and 74 suffer from error.

To eliminate such a problem, during the occurrence of a large variationin the angle of view, the state detecting block 69 not only sets theswitches 77 and 78 to the sides thereof to the memories 73 and 74,respectively, but also sets the switches 75 and 76 to the sides thereofto the memories 71 and 72, respectively. With this arrangement, thelow-reliability data are not input to the digital LPFs 61 and 62, andhigh-reliability data obtained prior to the occurrence of the largevariation in the angle of view and stored in the memories 71 and 72 areinput to the digital LPFs 61 and 62.

Regardless of the zooming information and the hand-shake information,the reliabilities of the amplitude γm and the initial phase Φmn areseparately determined. Reliability level information as a result ofdetermination is input to the state detecting block 69. If thereliability level information indicates that the reliability of theamplitude γm and the initial phase Φmn is low, the switches 75-78 areset to the sides thereof to the memories 71-74. In this way, past datahaving a high reliability is used.

<Advantages>

In accordance with the above-referenced embodiments, the flickerreduction operation is less sensitive to the external disturbance underthe illumination of the fluorescent lamp and under the illumination ofthe non-fluorescent lamp, and still provides fast response and trackingfeatures during the transitional state. The image pickup device performsthe proper flicker reduction operation smoothly without discordance atthe moment of the state transition, at the occurrence of the variationin the angle of view, or when the flicker detection parameter is low inreliability.

[Alternate Embodiments]

(Integration)

In each of the above-referenced examples, the input image signalIn′(x,y) is integrated throughout one line. Since the input image signalIn′(x,y) is integrated to obtain a sample value of the flicker componentwith the effect of a pattern of the image reduced, the integration maybe performed throughout a plurality of lines rather than only one line.One period of the fluorescent light flicker (on-screen flicker)appearing as a striped pattern on the screen corresponds to L(=M*60/100) lines as previously described. If at least two sample valuesare obtained in one period, i.e., L lines, the flicker component isdetected based on the sampling theorem.

In practice, a plurality of sample values, for example, at least 10sample values are preferably obtained from one period, i.e., L lines ofthe on-screen flicker. Even in this case, the horizontal period of theinput image signal In′(x,y) is integrated over a duration of time equalto or more than several times the horizontal periods, more particularly,ten times longer than the horizontal period. The integration time is notlimited to an integer multiple of horizontal periods, and for example,2.5 times the horizontal period is perfectly acceptable.

If the integration time is prolonged and a sample count per unit time isreduced, workload involved in the DFT operation by the DFT block 51 islightened. When a subject moves in a vertical direction of the screen,the effect of the motion is reduced.

(Other Alternate Embodiments)

In the primary color system of FIG. 2, the flicker components in the RGBprimary color signals is detected and reduced by the flicker reducers25R, 25G, and 25B on a per color signal basis. Alternatively, thepreviously described flicker reducer 25 is arranged on the output sideof the luminance signal Y of the synthesis matrix circuit 29 in order todetect and reduce the flicker component in the luminance signal Y.

In accordance with the above-referenced embodiments, the digital signalprocessor 20 including the flicker reducer 25 is implemented byhardware. Part or whole of the flicker reducer 25 or the digital signalprocessor 20 may be implemented by software.

In accordance with the above-referenced embodiments, the verticalsynchronization frequency is 60 Hz (with one field period equal to 1/60second). The present invention is applicable to a progressive typecamera having a vertical synchronization frequency of 30 Hz (with oneframe period being 1/30 second). In this case, three frame period ( 1/10second) is an integer multiple (the stripe pattern of the flicker is 10wavelengths over three frames) of the emission period of ( 1/100 second)of a fluorescent lamp, and the field of the above-referenced embodimentmay be substituted for the frame.

The present invention is applicable to an XY addressing type imagepickup element such as an image pickup element other than a CMOS imagepickup element.

INDUSTRIAL APPLICABILITY

In accordance with the present invention, the flicker unique to thefluorescent lamp appearing in the XY addressing type image pickupelement such as a CMOS image pickup element is detected and reducedreliably and sufficiently through the simple signal processing withoutusing a photosensitive element regardless of the subject, the videosignal level, and the type of the fluorescent lamp.

In particular, when the flicker reduction operation of the presentinvention is used for the RGB primary color signals, not only the lightand dark flicker but also the color flicker is accurately detected, andreliably and sufficiently reduced.

If the level of the signal falls within the saturation region, theflicker reduction is disabled. In this way, the signal component isprevented from being affected by the flicker reduction operation.

The flicker reduction operation is disabled under the illumination ofthe non-fluorescent lamp. In this way, the effect of the flickerreduction operation on the image quality is controlled.

If the subject changes greatly in a short period of time in response tothe operation or the action of the photographer, the immediatelyprecedingly estimated flicker component or the flicker componentestimated from the immediately preceding signal is calculated. In thisarrangement, flicker detection accuracy is free from degradation due alarge change in the subject happening in a short period of time.

The flicker reduction operation is disabled in the photographingcondition that does not need the flicker reduction operation, forexample, in the still image taking. In this way, the effect of theflicker reduction operation on the image quality is controlled.

The estimated flicker component is adjusted, and the adjusted flickercomponent and the input image signal are calculated. With thisarrangement, the signal component is robust to the effect of externaldisturbance under the illumination of the fluorescent lamp and under theillumination of the non-fluorescent lamp. The image pickup deviceachieves excellent response and tracking features at the transition.Furthermore, the image pickup device performs the processes smoothlywithout discordance at the state transitions, at the variation in theangle of view, or when the reliability of the flicker detectionparameters is low.

1. A flicker reduction method for reducing a fluorescent light flickercomponent in a video signal or a luminance signal obtained byphotographing a subject through an XY addressing type image pickupelement under an illumination of a fluorescent lamp, comprising: a stepof integrating the video signal or the luminance signal, as an inputimage signal, throughout a duration of time equal to or longer than onehorizontal period, a step of normalizing the integrated value or adifference value between the integrated values of adjacent fields oradjacent frames, a step of extracting a spectrum of the normalizedintegrated value or the normalized difference value, wherein a signalcomponent other than a flicker component is extracted as the normalizedintegrated value or the normalized difference value, a step ofestimating a flicker component from the extracted spectrum, wherein theestimated flicker component is approximated under the illumination ofthe fluorescent lamp to a predetermined order; wherein an amplitude andan initial phase of the estimated flicker component is estimated foreach order, and a step of performing a calculation operation on theestimated flicker component and the input image signal to cancel out theestimated flicker component.
 2. A flicker reduction method for reducinga fluorescent light flicker component in each of color signals of colorsobtained by photographing a subject through an XY addressing type imagepickup element under an illumination of a fluorescent lamp, comprising:a step of integrating the color signal of each color, as an input imagesignal, throughout a duration of time equal to or longer than onehorizontal period, a step of normalizing the integrated value or adifference value between the integrated values of adjacent fields oradjacent frames, a step of extracting a spectrum of the normalizedintegrated value or the normalized difference value, wherein a signalcomponent other than a flicker component is extracted as the normalizedintegrated value or the normalized difference value, a step ofestimating a flicker component from the extracted spectrum, wherein theestimated flicker component is approximated under the illumination ofthe fluorescent lamp to a predetermined order; wherein an amplitude andan initial phase of the estimated flicker component is estimated foreach order, and a step of performing a calculation operation on theestimated flicker component and the input image signal to cancel out theestimated flicker component.
 3. A flicker reduction method for reducinga fluorescent light flicker component in both a luminance signal andeach of color signals of colors, obtained by photographing a subjectthrough an XY addressing type image pickup element under an illuminationof a fluorescent lamp, comprising: a step of integrating each of theluminance signal and the color signal of each color, as an input imagesignal, throughout a duration of time equal to or longer than onehorizontal period, a step of normalizing the integrated value or adifference value between the integrated values of adjacent fields oradjacent frames, a step of extracting a spectrum of the normalizedintegrated value or the normalized difference value, wherein a signalcomponent other than a flicker component is extracted as the normalizedintegrated value or the normalized difference value, a step ofestimating a flicker component from the extracted spectrum, wherein theestimated flicker component is approximated under the illumination ofthe fluorescent lamp to a predetermined order; wherein an amplitude andan initial phase of the estimated flicker component is estimated foreach order, and a step of performing a calculation operation on theestimated flicker component and the input image signal to cancel out theestimated flicker component.
 4. The flicker reduction method accordingto claim 1, wherein the normalizing step comprises dividing thedifference value by the average value of the integrated values of aplurality of consecutive fields or consecutive frames.
 5. The flickerreduction method according to claim 1, wherein the normalizing stepcomprises dividing the difference value by the average value of theintegrated values of a plurality of consecutive fields or consecutiveframes, and subtracting a predetermined value from the resultingquotient.
 6. The flicker reduction method according to claim 1, whereinthe normalizing step comprises dividing the difference value by theintegrated value.
 7. The flicker reduction method according to claim 1,wherein the spectrum extracting step comprises Fourier transforming thenormalized integrated value or the normalized difference value.
 8. Theflicker reduction method according to claim 1, wherein it is determinedwhether a level of the input image signal falls within a saturationregion, and if it is determined that the level of the input image signalfalls within the saturation region, the input image signal is output asis as an output image signal.
 9. The flicker reduction method accordingto claim 1, wherein it is determined based on a level of the extractedspectrum whether the input image signal is from under the illuminationof the fluorescent lamp, and if it is determined that the input imagesignal is not from under the illumination of the fluorescent lamp, theinput image signal is output as is as an output image signal.
 10. Theflicker reduction method according to claim 1, wherein it is determinedwhether the subject changes greatly in a short period of time inresponse to an operation or an action of a photographer, and if it isdetermined that the subject changes greatly in a short period of time,one of an immediately precedingly estimated flicker component and aflicker component estimated from an immediately preceding signal, andthe input image signal are subjected to a calculation operation.
 11. Theflicker reduction method according to claim 1, wherein it is determinedwhether a photographing condition requires a flicker reduction operationand if it is determined that the flicker reduction operation isdetermined to be unnecessary, the input image signal is output as is asan output image signal.
 12. The flicker reduction method according toclaim 1, wherein the estimated flicker component is adjusted and theadjusted flicker component and the input image signal are subjected tothe calculation operation.
 13. The flicker reduction method according toclaim 1, wherein amplification data and initial phase data of theestimated flicker component are adjusted through respective low-passfilters, and a flicker component to be subjected to the calculationoperation together with the input image signal is generated using theadjusted amplification data and the adjusted initial phase data.
 14. Theflicker reduction method according to claim 13, wherein the adjustedamplitude data and the adjusted initial phase data are stored in amemory, and if a predetermined condition is detected, the flickercomponent to be subjected to the calculation operation together with theinput image signal is generated using the stored amplitude data and thestored initial phase data.
 15. An image pickup device comprising: an XYaddressing type image pickup element, means for integrating a videosignal or a luminance signal, as an input image signal, throughout aduration of time equal to or longer than one horizontal period, thevideo signal or the luminance signal being obtained by photographing asubject through the XY addressing type image pickup element, means fornormalizing the integrated value or a difference value between theintegrated values of adjacent fields or adjacent frames, means forextracting a spectrum of the normalized integrated value or thenormalized difference value, wherein a signal component other than aflicker component is extracted as the normalized integrated value or thenormalized difference value, means for estimating a flicker componentfrom the extracted spectrum, wherein the estimated flicker component isapproximated under the illumination of the fluorescent lamp to apredetermined order; wherein an amplitude and an initial phase of theestimated flicker component is estimated for each order, and means forperforming a calculation operation on the estimated flicker componentand the input image signal to cancel out the estimated flickercomponent.
 16. An image pickup device comprising: an XY addressing typeimage pickup element, means for integrating a color signal of eachcolor, as an input image signal, throughout a duration of time equal toor longer than one horizontal period, the color signal being obtained byphotographing a subject through the XY addressing type image pickupelement, means for normalizing the integrated value or a differencevalue between the integrated values of adjacent fields or adjacentframes, means for extracting a spectrum of the normalized integratedvalue or the normalized difference value, wherein a signal componentother than a flicker component is extracted as the normalized integratedvalue or the normalized difference value, means for estimating a flickercomponent from the extracted spectrum, wherein the estimated flickercomponent is approximated under the illumination of the fluorescent lampto a predetermined order; wherein an amplitude and an initial phase ofthe estimated flicker component is estimated for each order, and meansfor performing a calculation operation on the estimated flickercomponent and the input image signal to cancel out the estimated flickercomponent.
 17. An image pickup device comprising: an XY addressing typeimage pickup element, means for integrating each of a luminance signaland a color signal of each color, as an input image signal, throughout aduration of time equal to or longer than one horizontal period, thevideo signal and the luminance signal being obtained by photographing asubject through the XY addressing type image pickup element, means fornormalizing the integrated value or a difference value between theintegrated values of adjacent fields or adjacent frames, means forextracting a spectrum of the normalized integrated value or thenormalized difference value, wherein a signal component other than aflicker component is extracted as the normalized integrated value or thenormalized difference value, means for estimating a flicker componentfrom the extracted spectrum, wherein the estimated flicker component isapproximated under the illumination of the fluorescent lamp to apredetermined order; wherein an amplitude and an initial phase of theestimated flicker component is estimated for each order, and means forperforming a calculation operation on the estimated flicker componentand the input image signal to cancel out the estimated flickercomponent.
 18. The image pickup device according to claim 15, whereinthe normalizing means divides the difference value by the average valueof the integrated values of a plurality of consecutive fields orconsecutive frames.
 19. The image pickup device according to claim 15,wherein the normalizing means divides the difference value by theaverage value of the integrated values of a plurality of consecutivefields or consecutive frames, and subtracts a predetermined value fromthe resulting quotient.
 20. The image pickup device according to claim15, wherein the normalizing means divides the difference value by theintegrated value.
 21. The image pickup device according to claim 15,wherein the spectrum extracting means Fourier transforms the normalizedintegrated value or the normalized difference value.
 22. The imagepickup device according to claim 15, comprising means for determiningwhether a level of the input image signal falls within a saturationregion, and outputting the input image signal as is as an output imagesignal if it is determined that the level of the input image signalfalls within the saturation region.
 23. The image pickup deviceaccording to claim 15, comprising means for determining, based on alevel of the spectrum extracted by the spectrum extracting means,whether the input image signal is from under the illumination of thefluorescent lamp, and outputting the input image signal as is as anoutput image signal if it is determined that the input image signal isnot from under the illumination of the fluorescent lamp.
 24. The imagepickup device according to claim 15, comprising means for determiningwhether the subject changes greatly in a short period of time inresponse to an operation or an action of a photographer, and if it isdetermined that the subject changes greatly in a short period of time,causing the calculating means to subject to the calculation operation,one of a flicker component immediately precedingly estimated by theflicker component estimating means and a flicker component estimated bythe flicker component estimating means from an immediately precedingsignal, and the input image signal.
 25. The image pickup deviceaccording to claim 15, comprising means for determining whether aphotographing condition requires a flicker reduction operation and if itis determined that the flicker reduction operation is unnecessary,outputting the input image signal as is as an output image signal. 26.The image pickup device according to claim 15, comprising adjustingmeans for adjusting the flicker component estimated by the flickercomponent estimating means and generating the flicker component to besubjected to the calculation operation together with the input imagesignal.
 27. The image pickup device according to claim 15, comprisinglow-pass filter means for adjusting amplification data and initial phasedata of the flicker component estimated by the flicker componentestimating means, and generating the flicker component to be subjectedto the calculation operation together with the input image signal. 28.The image pickup device according to claim 27, comprising storage meansfor storing the amplitude data and the initial phase data, adjusted bythe low-pass filter means, and means for generating the flickercomponent to be subjected to the calculation operation together with theinput image signal using the stored amplitude data and the storedinitial phase data if a predetermined condition is detected.
 29. Aflicker reduction circuit for reducing a fluorescent light flickercomponent in a video signal or a luminance signal obtained byphotographing a subject through an XY addressing type image pickupelement under an illumination of a fluorescent lamp, comprising: meansfor integrating the video signal or the luminance signal, as an inputimage signal, throughout a duration of time equal to or longer than onehorizontal period, means for normalizing the integrated value or adifference value between the integrated values of adjacent fields oradjacent frames, means for extracting a spectrum of the normalizedintegrated value or the normalized difference value, wherein a signalcomponent other than a flicker component is extracted as the normalizedintegrated value or the normalized difference value, means forestimating a flicker component from the extracted spectrum, wherein theestimated flicker component is approximated under the illumination ofthe fluorescent lamp to a predetermined order; wherein an amplitude andan initial phase of the estimated flicker component is estimated foreach order, and means for performing a calculation operation on theestimated flicker component and the input image signal to cancel out theestimated flicker component.
 30. A flicker reduction circuit forreducing a fluorescent light flicker component in each of color signalsof colors obtained by photographing a subject through an XY addressingtype image pickup element under an illumination of a fluorescent lamp,comprising: means for integrating the color signal of each color, as aninput image signal, throughout a duration of time equal to or longerthan one horizontal period, means for normalizing the integrated valueor a difference value between the integrated values of adjacent fieldsor adjacent frames, means for extracting a spectrum of the normalizedintegrated value or the normalized difference value, wherein a signalcomponent other than a flicker component is extracted as the normalizedintegrated value or the normalized difference value, means forestimating a flicker component from the extracted spectrum, wherein theestimated flicker component is approximated under the illumination ofthe fluorescent lamp to a predetermined order; wherein an amplitude andan initial phase of the estimated flicker component is estimated foreach order, and means for performing a calculation operation on theestimated flicker component and the input image signal to cancel out theestimated flicker component.
 31. A flicker reduction circuit forreducing a fluorescent light flicker component in each of a luminancesignal and each of color signals of colors, obtained by photographing asubject through an XY addressing type image pickup element under anillumination of a fluorescent lamp, comprising: means for integratingeach of the luminance signal and the color signal of each color, as aninput image signal, throughout a duration of time equal to or longerthan one horizontal period, means for normalizing the integrated valueor a difference value between the integrated values of adjacent fieldsor adjacent frames, means for extracting a spectrum of the normalizedintegrated value or the normalized difference value, wherein a signalcomponent other than a flicker component is extracted as the normalizedintegrated value or the normalized difference value, means forestimating a flicker component from the extracted spectrum, wherein theestimated flicker component is approximated under the illumination ofthe fluorescent lamp to a predetermined order; wherein an amplitude andan initial phase of the estimated flicker component is estimated foreach order, and means for performing a calculation operation on theestimated flicker component and the input image signal to cancel out theestimated flicker component.
 32. The flicker reduction circuit accordingto claim 29, wherein the normalizing means divides the difference valueby the average value of the integrated values of a plurality ofconsecutive fields or consecutive frames.
 33. The flicker reductioncircuit according to claim 29, wherein the normalizing means divides thedifference value by the average value of the integrated values of aplurality of consecutive fields or consecutive frames, and subtracts apredetermined value from the resulting quotient.
 34. The flickerreduction circuit according to claim 29, wherein the normalizing meansdivides the difference value by the integrated value.
 35. The flickerreduction circuit according to claim 29, wherein the spectrum extractingmeans Fourier transforms the normalized integrated value or thenormalized difference value.
 36. The flicker reduction circuit accordingto claim 29, comprising means for determining whether a level of theinput image signal falls within a saturation region, and outputting theinput image signal as is as an output image signal if it is determinedthat the level of the input image signal falls within the saturationregion.
 37. The flicker reduction circuit according to claim 29,comprising means for determining, based on a level of the spectrumextracted by the spectrum extracting means, whether the input imagesignal is from under the illumination of the fluorescent lamp, andoutputting the input image signal as is as an output image signal if itis determined that the input image signal is not from under theillumination of the fluorescent lamp.
 38. The flicker reduction circuitaccording to claim 29, comprising means, under the control of externalmeans, for causing the calculating means to perform the calculationoperation on one of the flicker component immediately precedinglyestimated by the flicker component estimating means and the flickercomponent estimated by the flicker component estimating means based onan immediately prior signal, and the input image signal.
 39. The flickerreduction circuit according to claim 29, comprising means for outputtingthe input image signal as is as an output image signal under the controlof external means.
 40. The flicker reduction circuit according to claim29, comprising means for performing the calculation operation on theflicker component, estimated by the flicker component estimating meansand adjusted by the external means, and the input image signal.